Combining LSI and Vector Space to Improve Retrieval Performance

We describe an approach to information retrieval using a technique which combines two existing well-known information retrieval techniques: traditional vector space retrieval and Latent Semantic Indexing (LSI). The technique described in this paper assigns a query to document similiarity that is a weighted average of the similarity scores obtained by LSI and traditional vector space retrieval. Our approach improves retrieval performance by 12%, on average, over vector space retrieval on the seven collections studied. Four of the collections were chosen because their retrieval performance is higher when vector space retrieval is used instead of LSI, and our method improves retrieval performance by an average of 14% for these collections. Our work shows that LSI can be used to significantly improve retrieval performance on collections that previously did not benefit from LSI. We have also shown that a small, fixed LSI dimensionality reduction parameter (k=10) can be used to capture the ‘latent semantic’ information.